Crowdsourcing to support training for public health: A scoping review

Crowdsourcing is an interactive process that has a group of individuals attempt to solve all or part of a problem and then share solutions with the public. Crowdsourcing is increasingly used to enhance training through developing learning materials and promoting mentorship. This scoping review aims to assess the literature on crowdsourcing for training in public health. We searched five medical and public health research databases using terms related to crowdsourcing and training. For this review, the concept of crowdsourcing included open calls, designathons, and other activities. We used a PRISMA checklist for scoping reviews. Each full-text was assessed by two independent reviewers. We identified 4,071 citations, and 74 studies were included in the scoping review. This included one study in a low-income country, 15 studies in middle-income countries, 35 studies in high-income countries, and 11 studies conducted in multiple countries of varying income levels (the country income level for 12 studies could not be ascertained). Nine studies used open calls, 35 used a hackathon, designathon or other “a-thon” event, and 30 used other crowdsourcing methods, such as citizen science programs and online creation platforms. In terms of crowdsourcing purpose, studies used crowdsourcing to educate participants (20 studies), develop learning materials (17 studies), enhance mentorship (13 studies) and identify trainees (9 studies). Fifteen studies used crowdsourcing for more than one training purpose. Thirty-four studies were done in-person, 31 were conducted virtually and nine used both meeting options for their crowdsourcing events. Seventeen studies generated open access materials. Our review found that crowdsourcing has been increasingly used to support public health training. This participatory approach can be a useful tool for training in a variety of settings and populations. Future research should investigate the impact of crowdsourcing on training outcomes.


Introduction
In 2019, the World Health Organization (WHO) conducted a crowdsourcing open call to identify practical strategies to enhance research mentorship in low-and middle-income countries (LMICs) [1]. Open calls are an interactive form of crowdsourcing [2], which is a process that involves a group of individuals solving all or part of a problem, then sharing those solutions with the community [3]. The WHO open call solicited strategies to improve research mentorship and professional development, which were then assessed based on pre-specified criteria [1]. This open call received over 100 strategies, identified three individuals to contribute to a practical guide, engaged dozens of LMIC research institutions, and identified numerous open-access learning materials [1]. This underscores the ways that crowdsourcing approaches can enhance training and highlights the feasibility of crowdsourcing to enhance training engagement.
There is an increased recognition that we need to provide inclusive training to support diverse trainees [4]. There is a need to develop innovative approaches to identify early career investigators and nurture their opportunities for research [5], and to do so in participatory and inclusive ways [6]. Crowdsourcing approaches are one way to enhance training. Crowdsourcing has been previously used to identify LMIC researchers for training opportunities and engagement in health research as part of the WHO/TDR global programme [7,8]. Crowdsourcing approaches have also been used in various other learning contexts, including developing learning materials [9], identifying open-access training resources [10], and identifying ways to enhance public health education and mentorship [1,11,12]. Training programs may benefit from crowdsourcing approaches that enhance community engagement, spur innovation, and identify learners. Traditional training programs often involve experts delivering oneway instruction and guidance to trainees with the aim of enhancing their personal and professional development. Crowdsourcing can be used to shift from conventional training approaches to a more open and collaborative process. Instead of experts being primarily responsible for training methods and outcomes, a diverse group of individuals from the community can work together to frame training strategies. Crowdsourcing to support training can help to prepare public health practitioners for interdisciplinary partnerships and provide access to community-developed resources.
Despite the growing interest in the potential of participatory approaches such as crowdsourcing in promoting training, few studies have examined the application of crowdsourcing for public health training purposes. There is little comprehensive understanding of the characteristic components of this approach for training, as well as its best practices, outputs, and outcomes. Although there are several empirical articles on the use of crowdsourcing approaches to promote public health training, no efforts have been made to collate and synthesize this body of knowledge. Amidst the growing importance of innovation in public health training, crowdsourcing approaches could potentially provide innovative and participatory training modalities and components. Hence, this scoping review investigates and summarizes the extent to which crowdsourcing has been used to support and promote public health training and explores critical components of how crowdsourcing can be used to improve public health training.

Search strategy
We organized a scoping review of the literature, drawing on the framework of Arksey and O'Malley [13] and following the PRISMA Extension for Scoping Reviews (S1 Checklist). We registered the protocol for the scoping review in the Open Science Framework (Registration DOI: https://doi.org/10.17605/OSF.IO/Q3PNH). A scoping approach was selected given substantial differences in the training methods and outcomes, several different ways of using crowdsourcing that preclude pooling, and many gaps in the literature. On March 14 th , 2022, we conducted an initial search of five medical and public health research databases-PubMed, CINAHL, Embase, Global Health and Cochrane Library. We conducted a secondary search on April 5 th , 2023 to capture any new articles published during the year after our initial search. The search algorithm included variations of the following terms: crowdsourcing, hackathon, designathon, training, education, mentorship, and capacity building (S1 Text). We identified and adapted these search terms from prior crowdsourcing and training review literature. Included publications focused on using crowdsourcing methods and training. We used the WHO/TDR definition of crowdsourcing: "the process of having a large group, including experts and non-experts, solve a problem and then share the solution with the public [14]." This definition is grounded in a broader crowdsourcing set of approaches [15]. This includes open calls (also known as innovation challenges or contests), designathons (also known as hackathons or sprint-like events), and other forms of crowdsourcing. For this paper, we define training broadly to encompass formal education, informal education, mentorship, coaching, and capacity-building for a wide range of ages and learning backgrounds [16]. We exported records from our search, removed duplicates using EndNote X9, and performed online screening.

Study selection
Inclusion criteria were the following: relevant to public health; clear description of crowdsourcing methodology; the overall purpose was to enhance training, education, mentorship, or a related area. This included empirical descriptions of crowdsourcing training programs, clinical trials that used crowdsourcing methods for education or training, and descriptions of methods. There were no geographic or time restrictions on the search. Studies, commentaries/ editorials, and opinion pieces that described potential programs that have not been implemented were excluded. We excluded systematic, scoping, or narrative reviews and studies that were not written in English.
Two reviewers independently reviewed titles and abstracts for inclusion, and a third reviewer was available to resolve discrepancies. Then two independent reviewers examined full-text manuscripts, excluding studies based on the criteria above. Data were extracted about the purpose of crowdsourcing and health topic of the crowdsourcing event. We also extracted information on the following components of crowdsourcing that can support public health training: crowdsourcing method (open call, designathon, other), country income level, type of crowdsourcing event (digital, in-person or both), and whether open access materials were generated. We categorized crowdsourcing based on the Joint International Consensus Statement on Crowdsourcing [17]. Subsequently, a narrative synthesis of the extracted data was performed. The stages of our narrative synthesis included: 1) descriptive statistics to summarize the extent and nature of included studies and 2) thematic categorization, which involved identifying common training areas and grouping studies based on those training categories.

Identification of studies
We retrieved 3,438 publications from our initial database searches. After removing duplicates, the publications were screened for the relevance of the title and abstract, resulting in the exclusion of 2,108 publications. We further evaluated 88 publications for full-text eligibility. Of these, 28 articles were excluded for the following reasons: insufficient description of crowdsourcing methods (n = 8), not focused on crowdsourcing (n = 6), insufficient details on training, mentorship or education (n = 6), wrong article type (n = 3), not written in English (n = 1), and same crowdsourcing event already described in another study (n = 4). We retrieved 633 publications from our secondary database searches. The total number of publications retrieved from the initial and secondary database searches was 4,071. Overall, 74 studies were included in this review (Fig 1). Table 1 shows the descriptive characteristics of the 74 studies included in this review. The majority of studies were conducted in high-income countries (56.5%), followed by middle-

PLOS GLOBAL PUBLIC HEALTH
income countries (24.2%). Eleven studies (17.7%) used crowdsourcing approaches in multiple countries of varying income levels and one study (1.6%) was conducted in a low-income country. Thirty-five studies (47.3%) used a hackathon or other "a-thon" event as the crowdsourcing method. Thirty studies (40.5%) used crowdsourcing methods other than open calls, hackathons, or other "a-thon" events. These other methods mainly included online creation platforms, citizen science programs, and peer groups. The majority of these crowdsourcing events took place in-person (45.9%). Only 17 studies (23%) identified in this review generated open access materials. The majority of studies (81%) were published in 2018 or after.

Training areas
We identified the use of crowdsourcing to support four areas of public health training (Fig 2). Crowdsourcing was used to 1) educate participants, 2) develop learning materials, 3) promote mentorship, or 4) identify trainees. Below, we describe the role of crowdsourcing for each of these training areas in further detail.

Crowdsourcing to educate participants
Twenty studies (27%) used crowdsourcing to enhance participants' education and capacitybuilding (Table 2) [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37]. Of these studies, 12 were solely conducted in middle-or highincome countries. Twelve studies educated participants in-person, six were done virtually, and two used both methods. Eight studies used a hackathon or datathon to educate participants and five studies utilized citizen science programs. Hackathons were used to deliver a variety of educational content, including healthcare innovation [18], social work education [36], and neuroscience [20]. The Mount Sinai Health Hackathon, which is a 48-hour team-based competition that occurs annually, was a crowdsourcing event that serves as a model for team science education [23,26]. This hackathon brought together individuals from different disciplines to work on a shared health problem, which fostered an environment of experiential learning through collaboration and communication. Also, three of the five studies that used citizen science programs to crowdsource for education were focused on environmental health. These studies educated high school students on topics such as air pollution [21], radiation monitoring [24], and radon exposure [28]. Upon completion of the educational portion of the programs, these students engaged in citizen science by going into their communities and collecting and reporting environmental data.

Crowdsourcing to develop learning materials
Seventeen studies (23%) used crowdsourcing to develop learning materials and parts of health curricula (Table 3) [11,[38][39][40][41][42][43][44][45][46][47][48][49][50][51][52][53]. Of these studies, 11 were solely conducted in high-income countries. Fourteen studies only hosted the crowdsourcing event virtually. Ten studies made their learning materials publicly available. Nine studies utilized online creation platforms where participants collaborated to produce, review, collate and share learning materials. The use of online creation platforms resulted in the development of various learning resources, such as demonstration videos for learners on performing common pediatric procedures in   (Table 4) [1,8,9,13,[54][55][56][57][58][59][60][61][62]. Five of these studies were conducted in high income countries. Eight studies were conducted virtually, four were done in-person, and one used both methods to host their event. Eight studies used hackathons, three used open calls, and two used peer groups as their crowdsourcing method. All eight studies that organized hackathons provided mentors for their participants [8,9,54,55,58,[60][61][62]. The mentors had varying roles, which included providing expert knowledge, training participants in public speaking and presentation skills, reviewing ideas and prototypes, offering encouragement and support, and connecting participants to their networks. Two studies initiated open calls related to research mentorships; one solicited ideas to enhance research mentorship in LMICs [1] and another gathered data to understand and improve the impact of a global health research training program on trainees and students [13]. Peer groups were also vehicles to promote mentorship for internal medicine residents [56,57]. Peer mentorships were used to promote professional development and support resident well-being.

Crowdsourcing to identify trainees
Nine studies (12.2%) used crowdsourcing methods to identify talented trainees for further opportunities, making training programs more inclusive (  [70], how to improve the use of evidence-based practices in a behavioral health system [71]. In these past crowdsourcing open calls, participants with high-quality and promising ideas were invited to refine, finalize, present, and potentially implement their solutions through feedback and collaboration with content experts. These crowdsourcing methods were inclusive as they aggregated the ideas and perspectives of the trainees with the knowledge and experiences of the experts.

First Author Publication Year
Country Income Level*

PLOS GLOBAL PUBLIC HEALTH
13 of these studies included promoting research mentorship. One study used a crowdsourcing workshop in India to identify trainees and build their capacity to provide mental health services [76]. These trainees were identified after applying for the workshop and then paired with mentor experts who guided them through developing a mental health research funding proposal.

Discussion
This scoping review describes the extent and characterizes existing research on crowdsourcing for public health training. We found that crowdsourcing has been used to support four areas of training: to educate participants, develop learning materials, promote mentorship, and identify trainees. Studies in this review featured different crowdsourcing approaches to improve public health training. We found that these participatory approaches have supported training on a broad range of health topics, including environmental health [21,24,28], infectious diseases [11,25,54,55,60,69,70,74,75,84], and mental health [76]. This scoping review extends the literature on crowdsourcing to examine how it has been used to benefit public health training.
The findings of this scoping review provide an evidence base for the role of crowdsourcing to foster education and develop learning materials. These studies used a variety of crowdsourcing approaches such as hackathons, citizen science programs at schools, and online platforms to deliver education. These studies suggest that crowdsourcing can provide a structured method for community health and medical education. We also found that all studies that used crowdsourcing to develop learning materials used online collaboration systems. This demonstrates that crowdsourcing can be used to engage diverse online communities and provide a virtual environment where these communities can work together to develop training resources. Moreover, online crowdsourcing approaches may be less resource-intensive than in-person crowdsourcing events, which could be potentially useful in resource-limited settings. Crowdsourcing comes with an obligation to give back to the public who created the idea. Therefore, learning materials that were developed from crowdsourcing can then be used for future education and capacity-building. In the 13 studies that used crowdsourcing to develop learning materials, eight studies made their materials widely available to the public [11,40,41,43,48,49,52,53], which allows the resources to be accessible without restrictions.
We also found that crowdsourcing has been used to promote research and professional mentorship. A recurring theme among studies focused on promoting mentorship is the use of participatory approaches to increase community participation in research and development of interventions. Crowdsourcing events have been used to identify participants with promising ideas and connect them with mentors who can help them iteratively refine their ideas. Typically, this would include events where participants practice pitching their ideas to these mentors and receive tailored feedback. Also, an additional advantage that can arise from these crowdsourcing events is the opportunity for mentees to have access to their mentors' networks. This can help mentees strengthen their research ideas and broaden their own professional networks. It is also important for public health trainers to carefully consider and employ the most appropriate crowdsourcing method to enhance mentorship. Hackathons, for example, are a popular crowdsourcing approach to provide mentorship. Considerations for hackathons to successfully promote mentorship can include determining the appropriate mentormentee ratio for the event and incorporating designated and adequate mentoring sessions into the hackathon agenda.
Moreover, we found that crowdsourcing is a useful approach to identifying and engaging talented trainees. Open calls, in particular, can be used to select individuals for training. For example, one study used a crowdsourcing open call to solicit ideas on how to improve HIV self-testing among youth [70]. Participants with promising ideas were selected as finalists and proceeded to subsequent stages of the study for further development of their ideas and opportunities for apprenticeships. Another study used an open call to solicit ideas on improving the participation of women in an infectious disease research fellowship [69]. The top ideas from this open call were then implemented in the fellowship's next application cycle, which saw an increase in the number of women applicants. Open calls have the potential to reach a wide and varied audience, which can provide an opportunity for enhanced inclusivity and engagement of early investigators. These methods should be considered by public health researchers as suitable approaches to identify and engage with early investigators.
Our scoping review had several limitations. First, the crowdsourcing literature is diverse in form and content, making the pooling of studies for meta-analysis difficult. Second, we captured fewer studies from low-and middle-income countries. This may be the result of fewer crowdsourcing activities in those countries or less reporting of those experiences. Third, our scoping review did not include grey literature, thus relevant studies may have been missed. Fourth, we may not have captured some capacity-building programs that used crowdsourcing but were not formally evaluated.
Our findings have implications for public health research, programming, and education. Crowdsourcing can serve as an innovative model to advance public health training. Crowdsourcing provides a way to go beyond conventional didactic approaches to training to engaging and collaborative methods to training. This may result in public health professionals that are prepared to develop creative and novel solutions to address challenging public health issues. These participatory approaches can be considered for use by public health agencies looking to identify and provide funding opportunities for talented early investigators, educational institutions that are preparing public health students for the workforce, or organizations that seek to provide opportunities for professional development and mentorship.

Conclusion
Our scoping review found a wide range of studies supporting the use of crowdsourcing methods for training in public health. Future research should evaluate the impact of crowdsourcing on training outcomes.